Given the success with in-context learning of large pre-trained language models, we introduce in-context learning distillation to transfer in-context few-shot learning ability from large models to smaller models. We propose to combine in-context learning objectives with language modeling objectives to distill both the ability to read in-context examples and task knowledge to the smaller models. We perform in-context learning distillation under two different few-shot learning paradigms: Meta In-context Tuning (Meta-ICT) and Multitask In-context Tuning (Multitask-ICT). Multitask-ICT performs better on multitask few-shot learning but also requires more computation than Meta-ICT. Our method shows consistent improvements for both Meta-ICT and Multitask-ICT on two benchmarks: LAMA and CrossFit. Our extensive experiments and analysis reveal that in-context learning objectives and language modeling objectives are complementary under the Multitask-ICT paradigm. In-context learning objectives achieve the best performance when combined with language modeling objectives.

This paper presents a subsampling-task paradigm for data-driven task-specific experiment design (ED) and a novel method in populationwide supervised feature selection (FS). Optimal ED, the choice of sampling points under constraints of limited acquisition-time, arises in a wide variety of scientific and engineering contexts. However the continuous optimization used in classical approaches depend on a-priori parameter choices and challenging non-convex optimization landscapes. This paper proposes to replace this strategy with a subsampling-task paradigm, analogous to populationwide supervised FS. In particular, we introduce JOFSTO, which performs JOint Feature Selection and Task Optimization. JOFSTO jointly optimizes two coupled networks: one for feature scoring, which provides the ED, the other for execution of a downstream task or process. Unlike most FS problems, e.g. selecting protein expressions for classification, ED problems typically select from highly correlated globally informative candidates rather than seeking a small number of highly informative features among many uninformative features. JOFSTO's construction efficiently identifies potentially correlated, but effective subsets and returns a trained task network. We demonstrate the approach using parameter estimation and mapping problems in clinically-relevant applications in quantitative MRI and in hyperspectral imaging. Results from simulations and empirical data show the subsampling-task paradigm strongly outperforms classical ED, and within our paradigm, JOFSTO outperforms state-of-the-art supervised FS techniques. JOFSTO extends immediately to wider image-based ED problems and other scenarios where the design must be specified globally across large numbers of acquisitions. Code will be released.

本文涉及分割中的伪标记。我们的贡献是四倍。首先，我们提出了伪标签的新表述，作为一种预期最大化（EM）算法，用于清晰的统计解释。其次，我们纯粹基于原始伪标记，即Segpl，提出了一种半监督的医学图像分割方法。我们证明，SEGPL是针对针对2D多级MRI MRI脑肿瘤分段任务和3D二进制CT肺部肺血管分段任务的半监督分割的最新一致性正则方法的竞争方法。与先前方法相比，SEGPL的简单性允许更少的计算成本。第三，我们证明了SEGPL的有效性可能源于其稳健性抵抗分布噪声和对抗性攻击。最后，在EM框架下，我们通过变异推理引入了SEGPL的概率概括，该推论学习了训练期间伪标记的动态阈值。我们表明，具有变异推理的SEGPL可以通过金标准方法深度集合在同步时执行不确定性估计。

我们提出了Prosub：渐进式采样，这是一种基于深度学习的自动化方法，该方法是一个过采样的数据集（例如，多通道的3D图像），信息损失最小。我们以最近的双NETWORK方法为基础，该方法赢得了MICCAI多扩散（MUDI）定量MRI测量测量取样重建挑战，但通过在艰难的决策边界进行下采样，遭受了深度学习训练的不稳定。 Prosub使用递归功能消除（RFE）的范式，并在深度学习训练期间逐步进行亚子样本测量，从而提高优化稳定性。 Prosub还集成了神经体系结构搜索（NAS）范式，从而允许网络体系结构超参数响应亚群采样过程。我们显示，Prosub优于Mudi Miccai挑战的获胜者，在MUDI挑战子任务和对临床应用有用的下游过程的定性改进方面产生了> 18％的MSE。我们还展示了合并NAS并分析Prosub组件的效果的好处。由于我们的方法概括了除MRI测量选择重建之外的其他问题，因此我们的代码是https://github.com/sbb-gh/prosub

准确的多级细分是医学成像的长期挑战，特别是在课程份额强烈相似的情况下。视网膜照片中的视网膜血管是一种这种情况，其中需要识别和静脉互相鉴定和静脉曲张。分段内分类，即静脉归类为动脉或反之亦然，经常发生在动脉和静脉相交时发生，而在二进制视网膜血管分割中，误差率低得多。因此，我们提出了一种新的方法，可以将多类分段分解为多个二进制文件，然后是二进制到多级融合网络。该网络合并动脉，静脉和多级特征图的表示，每个地图都受到对抗培训中的专家船舶注释的监督。基于跳过连接的合并过程明确地维护了特定于类的梯度，以避免在深层中消失的梯度，以支持歧视特征。结果表明，与在Drive-AV，LES-AV和HRF上的三种最新的深度学习方法相比，我们的模型分别将F1分数提高了4.4 \％，5.1 \％和4.2 \％ -av数据集。代码：https://github.com/rmaphoh/learning-avegundation.

Recent studies have shown that using an external Language Model (LM) benefits the end-to-end Automatic Speech Recognition (ASR). However, predicting tokens that appear less frequently in the training set is still quite challenging. The long-tail prediction problems have been widely studied in many applications, but only been addressed by a few studies for ASR and LMs. In this paper, we propose a new memory augmented lookup dictionary based Transformer architecture for LM. The newly introduced lookup dictionary incorporates rich contextual information in training set, which is vital to correctly predict long-tail tokens. With intensive experiments on Chinese and English data sets, our proposed method is proved to outperform the baseline Transformer LM by a great margin on both word/character error rate and tail tokens error rate. This is achieved without impact on the decoding efficiency. Overall, we demonstrate the effectiveness of our proposed method in boosting the ASR decoding performance, especially for long-tail tokens.

Deep learning classifiers provide the most accurate means of automatically diagnosing diabetic retinopathy (DR) based on optical coherence tomography (OCT) and its angiography (OCTA). The power of these models is attributable in part to the inclusion of hidden layers that provide the complexity required to achieve a desired task. However, hidden layers also render algorithm outputs difficult to interpret. Here we introduce a novel biomarker activation map (BAM) framework based on generative adversarial learning that allows clinicians to verify and understand classifiers decision-making. A data set including 456 macular scans were graded as non-referable or referable DR based on current clinical standards. A DR classifier that was used to evaluate our BAM was first trained based on this data set. The BAM generation framework was designed by combing two U-shaped generators to provide meaningful interpretability to this classifier. The main generator was trained to take referable scans as input and produce an output that would be classified by the classifier as non-referable. The BAM is then constructed as the difference image between the output and input of the main generator. To ensure that the BAM only highlights classifier-utilized biomarkers an assistant generator was trained to do the opposite, producing scans that would be classified as referable by the classifier from non-referable scans. The generated BAMs highlighted known pathologic features including nonperfusion area and retinal fluid. A fully interpretable classifier based on these highlights could help clinicians better utilize and verify automated DR diagnosis.

Existing natural language understanding (NLU) models often rely on dataset biases rather than intended task-relevant features to achieve high performance on specific datasets. As a result, these models perform poorly on datasets outside the training distribution. Some recent studies address the above issue by reducing the weights of biased samples during the training process. However, these methods still encode biased latent features in representations and neglect the dynamic nature of bias, which hinders model prediction. We propose an NLU debiasing method, named debiasing contrastive learning (DCT), to simultaneously alleviate the above problems based on contrastive learning. We devise a debiasing positive sampling strategy to mitigate biased latent features by selecting the least similar biased positive samples. We also propose a dynamic negative sampling strategy to capture the dynamic influence of biases by employing a bias-only model to dynamically select the most similar biased negative samples. We conduct experiments on three NLU benchmark datasets. Experimental results show that DCT outperforms state-of-the-art baselines on out-of-distribution datasets while maintaining in-distribution performance. We also verify that DCT can reduce biased latent features from the model's representations.

对使用基于深度学习的方法来实现正电子发射断层扫描（PET CT）扫描中的病变的完全自动分割的研究兴趣越来越多，以实现各种癌症的预后。医学图像细分的最新进展表明，NNUNET对于各种任务是可行的。但是，PET图像中的病变分割并不直接，因为病变和生理摄取具有相似的分布模式。它们的区别需要CT图像中的额外结构信息。本文引入了一种基于NNUNET的病变分割任务的方法。提出的模型是根据关节2D和3D NNUNET结构设计的，以预测整个身体的病变。它允许对潜在病变的自动分割。我们在AUTOPET挑战的背景下评估了所提出的方法，该方法衡量了骰子评分指标，假阳性体积和假阴性体积的病变分割性能。

深Q学习网络（DQN）是一种成功的方式，将增强学习与深神经网络结合在一起，并导致广泛应用强化学习。当将DQN或其他强化学习算法应用于现实世界问题时，一个具有挑战性的问题是数据收集。因此，如何提高数据效率是强化学习研究中最重要的问题之一。在本文中，我们提出了一个框架，该框架使用深q网络中的最大均值损失（m $^2 $ dqn）。我们没有在训练步骤中抽样一批体验，而是从体验重播中采样了几批，并更新参数，以使这些批次的最大td-Error最小化。所提出的方法可以通过替换损耗函数来与DQN算法的大多数现有技术结合使用。我们在几个健身游戏中使用了最广泛的技术DQN（DDQN）之一来验证该框架的有效性。结果表明，我们的方法会导致学习速度和性能的实质性提高。